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ABSTRACT 

A follow-up study of fraduates from St. Mary's Junior 
College (Minnesota) associate degree nursing program is used to 
demonstrate a multi-trait, multi-method basis for arriving at faculty 
judgments about the outcomes of a program. The follow-up study 
attempted to determine: (1) graduates' and employing agencies! 
perceptions of competencies four months following graduation; (2) 
characteristics of the graduates nursing careers and employment 
patterns; and (3) the manner in which graduates were utilized in 
nursing practice. The findings used evaluation matrices and 
statistical techniques to compare graduate self-evaluation and 
employer evaluation, college grade point averages, State Board 
composite and test scores, nursing grade point avérages, and 
practices and types of employment. Faculty then reviewed the 
non-cengruent areas which were revealed by the evaluation. The data 
obtained indicated (1) congruence between the intents or objectives 
of the program and evaluative judgments by nursing service personnel; 
(2) college GPA, nursing GPA, and State Board scores did not predict 
the employers’ ratings of the graduates; and (3) a tendency for ~ 
graduates with higher GPAs and higher State Board scores to be rated 
higher by the employer. (RN) 
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FOLLOW -UP STUDIES IN NURSING: 


A CASE FOR DETERMINING WHETHER PROGRAM OBJECTIVES ARE ACHIEVED* 


a 


INTRODUCTION 


The purpose of: this article is to identify a methodology for 
evaluating innovative nursing programs, It is assumed that penwink 
methods of curriculum evaluation are based on traditional outcomes 
such as performance on state boards and nurcing achievement tests. 
Traditional standards used for evaluation are inadequate when applied 
to innovative curricula, This inadequacy is based upon the differ- 
ences in intents of various curricula. Therefore, the standards used 
to arrive at juiganies past to reflect congruence or consistency with 


. 


the particular curriculum intents. 
HISTORICAL BACKGROUND 


A follow-up study of graduates from St. Mary's Junior College 


(SMJC) Associate degree Nursing Program, Minneapolis, Minnesota, will 


be used to illustrate the development of a rational; evaluation plan, 


The follow-up study is an example of action research which supplements 


the on-going curricular evaluation. The following background 


*We wish to acknowledge that both the curriculum and follow-up 
study discussed in this paper were group efforts of St. Mary's Junior 
College Nursing Faculty. . However, the authors assume responsibility 
for the content of this paper. 
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information illustrates. one faculty's concern with evaluating its 
graduates according to the philosophy and goals of its program. 

The curriculum organization and content for the Nursing program 
were determined during a six-week faculty workshop held prior to the 
junior college . pening in September, 1964. As a framework for the 
nursing curriculum, the nursing faculty decided to use Abraham 
Maslow's hierarchy of needs. It was felt that to begin with an 
understanding of the normal needs of healthy persons was essential 
to a later understanding of emotional and pliysical illness, The 
program became one with seven sequential courses in nursing. Each 
course was one quarter in length although one quarter was an 
accelerated summer session, 

As students in the first class progressed in meeting the 
objectives of each course, they learned to tine a problem-solving 
approach to patient care. At the same time, students studied the 
needs of human beings and interferences ill or hospitalized people 
encounter in meeting needs independently. ’ 

There were two primary reasons for undertaking a follow-up 
study of the graduates. To the knowledge of the faculty, an 
associate degree curriculum based on a hierarchy of normal needs 

. 


had not been developed in nursing. Therefore, these graduates should 


be studied because their curriculum was unique. The curriculum 


represented an experiment in method, time, and odjectives which 


required evaluation to ascertain whether the graduates were competent 


in the practice of technical nursing. Secondly, it was felt that 


TO a ee ee eae 
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valid predictions for students in a particular program arise from 
evaluation of the graduate of that program, Previous predictions for 
the success of St. Mary's graduates were based on a different 
curriculum (three-year diploma) and were not applicable to graduates 
of the associate degree program. Other reasons for conducting a 
follow-up study were to determine the utilization and employment 
patterns of these graduates. 

Based on the previous considerations, the nursing faculty 
established a committee -to study its first class of graduates in 1966, 
After review of the literature on curriculum evaluation, employment, 
and utilization of associate degree nursing graduates, the following 
purposes were identified for the follow-up study: 

L. To determine by means of a questionnaire the stationtne® ‘ 


and the employing agencies' perceptions of competencies 
four months following graduation. 


To determine the characteristics of the graduates' 
nursing careers and employment patterns. : 


To determine the manner in which the graduates were 
utilized in nursing practice, 


Development _of Behavioral Statements 


The goals or intents for graduates of the program were 
developed by the faculty and stated in a paper titled, "Competencies 
of the Nursing Program Graduates of St. Mary's Junior College." The 
paper was developed to first introduce the reader to the philosophy 
and objectives of the SMJC Nursing program. Then the paper went on 


to illustrate ‘the performance of the graduates which could be expected 
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in selected areas of patient care, In seviving at statements of com- 
petency, all course objectives and areas of study were reviewed and 
the faculty arrived at a stun bitch they felt would have been 
“achieved by all students successfully completing the program in 
nursing. The faculty used the statement of competencies in several 
ways. Foremost, the statement was used as a presentation of the 
minimal hancaivan behaviors characteristic of these graduates which 
. reflected the faculty's philosophy of associate degree nursing educa- 
tion, Furthermore, the statement was used as a tonl to facilitate 
communication between faculty, graduates, the public, and/or employing © 
heal'th care agencies. Since these graduates were among the first 
associate degree nurses prepared in Minnesota, it was ivi to 
‘communi.ate how these graduates could best be utilized and what they 


would be competent to do in the health care field, For purposes of 


evaluation, an important pse of the competencies was preparation of 


the employing agencies for the follow-up study. 
, 


Rationale 


In innovative cuecidile develapunne and evaluation, it is 
important to use a plan which incorporates both on-going ana summative 
evaluation. Stake's approach to curriculum evaluation requires a 
curriculum rationale upon which two evaluative matrices can be 
established--one for description and the second for judgement. The 


. 
formal curriculum evaluation plan thus takes the following form:2 
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Rationale Intents Observations Standards Judgments 
e 
Antecedents 
Transactions Le 
Outcomes 
Description Matrix Judgment Matrix 


In using the above evaluation plan, two phenomena are of 
concern as one gathers data. First, the curriculum evaluator 
searches for congruence across in enondie: ik a horizontal level. 
Secondly, the evaluator looks for logical and/or aapictesl contingency 
between categories on a vertical leve]., In this manner, it is the 
congruence and contingency between areas of the matrices which lead to 
such final judgments as whether or not the particular educational 


program is accomplishing what it intended to accomplish, 


For purposes of this paper, the aspects of on-going evaluation 


from Stake's model will not be discussed. However, on-going eva lua- 
tion of contingencies between intents, transactions, and outcomes 
were carried out during the period of the students’ educational 
expebtsace. In this paper, aspects of summative evaluation from 
Stake's model will be discussed as depicted by the follow-up study, 
State Board of Nursing examination scores, and final grade suit 
éverages. This provides for a milti-trait, multi-method basis for 


judgment of congruence between intents and outcomes, 


° 
te 
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Population 


Because it vas a manageable number (128 graduates in nursing), 
all graduates were included in the study. All subjects were admitted 
in September, 1964, “at the opening of St. Mary's Junior College and 
graduated in June, 1966, with a major in nursing, Age, sex, previous 
ction experiences, and working experiences were not controlled. 
Ability and past academic achievement were controlled to the extent 
that the admission standards of the college included a range of high 
school ranks from the fiftieth percentile to the ninety-ninth per- 
centile, and a range of American College Testing battery composite 4 
scores from the fortieth percentile to the ninety-ninth percentile. 
However, these admission standards were not rigidly applied to this 


class. - ‘ 


Questionnaires 

Two questionnaires were developed for the follow-up study. 
One questionnaire was sent to the graduates of the program four months 
following graduation. This questionnaire included requests for basic 
data about the individual, an employment history, and information 
about the orientation to the first position taken. The final portion 
of the questionnaire sent to graduates asked that the erahustes rate 
themselves on a four-point scale as to the proficiency with which they 
performed selected nursing competencies, The competencies rated in 


the self-evaluation were taken from the paper describing the abilities 


of graduates. 


Sita Lae 
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A second. questionnaire was developed for completion by the 


employer of the nursing graduate. This questionnaire was identical to 


the final portion of tie questionnaire sent to the graduates. The 
health care agency employing the ‘graduates was asked to have a nursing 
service person, working closely with thesgraduate, rate him on com- 
petencies at four months following graduation. 

Both questionnaires, listed fifty-eight competencies selecead 
as representative of the curriculum. Following is an example of a 
selected coupatency that respondents were asked to rate: 


When giving direct patient: care, she has used ‘the following 
steps of the problem-solving process: 


a. observation of patient behavior 


b, definition of the nursing problem 


c. search for additional facts 
d, redefinition or refinement of the nursing problem 


e. selection of reine solutions according tc established j 
criteria % : i 


¢ y 


f£. establishment of criteria for evaluation of nursing. 1 : 
solutions 6. * 
g. implementation of sélected nursing solutions 
® . g. 
h. evaluation of nursing solution according to criteria : ) 
established for evaluation 


Findings 

The descriptive data from depnive graduate questionnaires 
concerning employment history; types of positions, responsibilities, 
and agency orientation were tabulated and percentages were obtained, “ 


Din qreeerers> 


- 
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The graduates’ who sought staff nursing positions were able to find 


~ 


employment; 97 per cent™of the respondents were employed four months 


- ’ sd A 
. following graduation and 91 per cent of the sixty-nine respondents 


‘reported hospital staff nurse positions. The respondents’ functions 


and responsibilities were primarily in the area of patient Gate: 

(88 per cent), although 37 per cent reported team leadership responsi- 
bilities and 25 per-cent reported charge nurse Saaponatbtiities: The 
stated competenciés of graduates specified that they Seen not pre=, 
pared to function as team Leaders or charge nurses without additional 
orientation and preparation from nursing service personnel, It is 
probable that graduates were being asked to perform such functions 
without adequate orientation. In terms of general orientation, the 
respondents felt they received desired help from nursing service in 
the areas of direct patient care, organization of work; leadership 
pecivitias,eoteil tetment of priorities in patient care, and decision 


* 


making. 


As stated earlier, the renee were asked to rate themselves 
» 


_at four months after graduation on their proficiency of performing the ~ 


"+ 


specified competencies, The graduates' employers were asked to rate 
hia at four months after graduation on their proficiency of perform- . 
ing the faa specified competencies, The graduates and the empioyers 
used a rating scale ae tee difficulty, some difficulty, grea€ diffi- 
culty, or no opportunity. . Percentages of responses were tabulated 

in each area and discrepancies between responses .of graduate and 


employer were calculated.. There were sixty-nine graduate respondents 


~ = F PI 
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and 101 employer respondents. Only the fiftyseight graduate and 
employee ratings which corresponded were used when calculating the 
discrepancy of responses of the employer from the graduate, Each of 
the fifty-eight competencies was analyzed separately. Only some of 
the findings will be summarized for the purpose of this paper. 


In their rating of 101 graduates, the employers indicated that 


at least 10 per cent of the graduates had "great difficulty" perform- 


ing eleven ss fifty-eight competencies, In their rating of 


themsélves, at least 10 per cent of the sixty-nine graduates indicated 


that they had "great difficulty" in performing five of fifty-eight. 


competencies, Only one of the preceding sixteen competencies were 


similarly rated by both groups. 


In their self-rating, at least °20 per cent of the graduates 
taateatod that they had "no opportunity" to perform eight of, the 
fifty-eight Sompgeencsee; however, 101 employers indicated that at ~ 
least 20 per cent of the graduates had "no dpportunity" to perform 
sixteen ee the fifty-eight competencies, ekéy competency rated as 
such by ‘is graduates was rated sinilarly by the employers. 


In the analysis of the discrepancy between the ee ag of 
@ 


: fifty-eight graduates and the eral at least 20 per cent of the. 


graduates overrated: thenselves by one ‘ aciay scale on fineteen compe-= 
téncies. At Teast 20 fer ‘cent underratdd chemsalven on ten competen- 


mee when compared “to the employer rating. S a 


The information summarized in the Unie bas represents 


an at tempb-toofind napa “éE the divect: measures of progtam 


> & 
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intents and usual indirect measures. Tables 1 and 2 summarizé the 
relationship between the employers' average rating of competency and 
the nursing grade point average and the State Board of Nursing : 

: on af 
_Examination scores. Inspection of both relationships indicates a 0 | 
trend toward Migher rating of competence and higher grade point 
averages and State Board composite scores. , 

ts 7 
Table 1, Comparison of Graduates’ Nursing Grade Point Average and 
: Employers' Average Rating for Each Graduate. 
\ ; : 6 
Employers' Number 

Nursing GPA Average Rating of Competency* | Graduates 

‘ P ¢ ’ 
Less than 2.00 1.60 :' 7 ° 

- foe | 
2.00 - 2.49 ’ be ; 22 
2.50-2.99 “% | 1.37 19 

| ' ; 

3.00 - 3.49. : 1.33. - 6 
3.50- 4.00 * 11.30 3% , 


*In scoring the employers’ rating "no difficulty" equals 1. "Some 
difficulty" equals 2. "Great difficulty" equals 3; .The scores, for 
each graduate on each specified behavior were added and then 
averaged. ‘ . ' 


#kData of one graduat2 were bEimina ted. 


The prce2uct-moment correlation was used to analyze the- 


¢ 


relationships between the nursing grade point average and the State 


Board Examination scores, college gra’e point average, the employers' 


average rating of competency. Sidi lar analysis was done to compare the 


4 
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Table 2. Comparison. of State Board Composite Scores with Employers' 
Average-Rating. 


SS a en ee ——————e 
State Board Employers' Number of 
Composite _ Average Rating of Competency* Graduates 
200's , 1.62 
300's 1,52 
4600's 1.46 
o's 
600's 
“~~ ¥*The same procedure was followed in arriving at the employers' 
“average rating as. reported in Table 1. 


*kData of one graduate were eliminated, 


7 


itoteea' average rating of competency and the State Board Composite 

, score, ‘Tt may Soa tae the relationship between the nursing grade 
point average, the State Board Examination scores, and college grade 
point average were statistically sipiittcaat at the 0.01 level. The 
employer s' average rating of competency did not relate significantly 
with either the nursing grade point average or the State Board 


-composite, This analysis is summarized in Table 3. 
DISCUSSION 


The previous presentation of findings combined information 


obtained by self-evaluation and employe® evaluation with other data 


such as the college: grade point averages, State Board scores, and 


7 


7 ~ 


? 
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Table 3. Comparison of Nursing Grade Point Average, State Board 
Scores, College Grade Point Average, and Employers' 
Average Rating. ; 


higlayers’ Average 
N-58 Nursing GPA Rating of Competency 
State Board Coupoxite +.58 (.01) -.14 (n.s.) 
State Board Medical +.57 (.01) 
State Board Surgical +.54. (.01) 
State Board Obstetrics +.55 (.01) 
, State Board Pediatrics +.57 (.01) 
State Board Psychiatric | +.54 (.01) 
bitters GPA +.94 (.01) 


Employers' Average 
Rating of Competency -.1l (n.s.) 


nursing grade point average. Such a collection of data provides a 
multi-trait, multi-method basis for the faculty judgments about tha 
outcomes of an innovative program, 

It is in the judgments phase that faculty seek the existence 
of congruence kacuwet the program's intents, observations,and stan- 
dards which lead to evaluative decisions abcut a particular program. 


The process of, curriculum innovation seems inherently endowed with 


internal and external conflict and criticism. However, changes in 


innovative programs are best made on the basis of data concerning the 


‘program intents. The use of matrices such as the ones proposed by 
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Stake are useful guides for faculty to syateuabteatiy collect data for 
decision-making about curriculum revision and/or continuation, 

The data obtained fro and about the graduates concerning 
their employment history and practice were quite similar to the data 
expected and found in the research literature on utilization and 
practice of associate degree nursing graduates. From studying the 
previous tables, it can be seen that the college grade point average, 
the nursing grade point dvicage, and the State Board scores did not 
predict the employers’ ratings of the graduates. There was a tendency 


for graduates with higher grade point averages and higher State Board 


. scores to be rated higher by the employer. It was felt that the data - 


obtained demonstrated congruence between the intents or objectives of 
the program and evaluative judgments by nursing service personnel. 

Overall, when analyzing the results of the data obtained in 
the follow-up study, the faculty considered several limitations of 
this study, For example, only°54 per cent of the graduates responded 
to the questionnaires, Follow-up procedures, such as ‘sacond mailings 
and telephone contacts, did not succeed in achieving the 60 per cent 
minimum desired return rate. It is difficult to predict the effect 
on the data that would occur had the remaining 46 per cent of the 
graduates responded, Likewise, when the employer responded "no 
opportunity," it is difficult to know whether he meant the graduate 
had "no opportunity" to perform the competency or if the employer had 
no opportunity to observe the graduate performing the competency. 


Furthermore, it is difficult to be certain that the graduate and 


_« _Agses bs . 
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employer interpreted the competencies with the same meaning. 

‘The faculty felt that the questionnaires used in this study 
required continued refinement in order to arrive at statements of 
competency which were more similar in meaning to nursing service 
personnel and graduates, In addition, it was felt that increasing 
the number of possible ratings on a competency would allow greater 
discrimination between graduates, 

It was determined that the follow-up study achieved a measure 
of the graduates’ perception of competency and the employers" per- 
ception of graduate competency four months fullowing graduation, 
These results held greatest relevance for faculty decisions concern- 
ing curriculum revision and/or continuation, 

The most carefully reviewed areas were the eleven out of 
fifty-eight behaviors which employers indicated graduates had "great 
difficulty" and the five out of fifty-eight behaviors with which the 
graduates felt they had difficulty. For example, at least 10 per 
cent of employers’ ratings indicated "great difficulty" in seven 
statements relating to the use of problem solving in approaching 
nursing care. This may have indicatéd”’that the individuals -rating 
the graduates did not understand these behaviors, that the graduates 
had difficulty using problem solving in practice, or had difficulty 
interpreting such an approach to supervisory personel. That at 
least 10 per cent of the employers rated the graduates as having 
"great difficulty" deterniitus: oatienta’ perceptions may have indi- 


cated that the statement of behavior was too vague or difficult to 
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measure, especially since this difficulty was not reported by the 
graduates. The fact that employers reported that at least 10 per 
cent of the ‘graduates had "great difficulty" giving information to 
patients and relatives was difficult to evaluate. The eetee cay have 
meant that the graduates had difficulty explaining such aspects as 
laboratory tests or more extensive patient-teaching situations. 
Reporting that graduates had "great difficulty" participating in the 
referral of patients to public health agencies may indicate a‘nursing 
situation in which continuity of care between acute hospitals and 
public health agencies is not achieved, The rating might also’ have 
meant that to some extent the objectives of the curriculum were 
realized in that the graduates were not-prepared to design long- 
range care or make nursing diagnoses for the patient at some stages 
of illness, 

’ The report by graduates that at least 10 per cent of the 
graduates had "great difficulty" evaluating the adequacy or inade- 
quacy of patients’ diets, providing care to the dying patient and 
his relatives, and understanding the laboratory tests commonly done 
to determine fluid and electrolyte balance concerned the faculty. The 
faculty questioned whether the on-going curriculum revision which had 
taken place would ensure that later students sould perform more 
competently. The faculty also considered whether the statements of 
competency were appropriate for the beginning technical nurse, 


It was found difficult to interpret those competency state- 


ments in which the gradvates overrated or underrated themselves in 
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relation to the employers' rating. A concept felt important for 
faculty discussion was evaluation of student performance in the 
laboratory ‘or clinical area, For example, did the laboratory 
evaluations achieve congruence between teacher and student rating of 


performance? tf 
SUMMARY 


This article attempted to present an approach to the evalua- 
tion of an innovative program in nursing, The development of nursing 
Programs requires that care be taken to evaluate the graduates on the 
basis of program goals and/or objectives which are derived froma 


stated rationale for the program. 


Aspects of Stake's model were used to demonstrate evaluation 


matrices considered in the evaluation of graduates of a new nursing 
‘program, Faculty reviewed the non-congruent areas revealed by the 
evaluation, Selected findings were reported comparing graduate self- 
evaluation and employer evaluation, college grade point averages, 
State Board composite and test scores, nursing grade point averages, 
practices and types of employment. The study demonstrates a multi- 
trait, multi-method basis for arriving at faculty judgments about the 


outcomes of a program, 
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FOOTNOTES 


For an account of the development of the SMJC program, see: 


From Diploma School to College: Two Case Studies on Changing 


Patterns Within Institutions for Nursing Education. Rochester, 
New York: National Commission for the Study of Nursing and 


Nursing Education, 1971. Lysaught's report describes the 
transition from a three-year diploma program to a two-year 
associate degree program in nursing. 


Robert E. Stake, The Countenance of Educational Evaluation, 
p. 350. In, Anderson, Faust, Roderick, Cunningham and Andre. 


Current Research on Instruction, Englewood Cliffs, New Jersey: 


Prentice-Hall, 1969, 


